本文将介绍如何通过Sentinel实现Redis集群(主从)的高可用方案,该方案需要使用Jedis2.2.2及以上版本(强制),Redis2.8及以上版本(可选,Sentinel最早出现在Redis2.4中,Redis2.8中Sentinel更加稳定),同时将redis与spring-date-redis集成。
一、Sentinel介绍 Sentinel是Redis的高可用性(HA)解决方案,由一个或多个Sentinel实例组成的Sentinel系统可以监视任意多个主服务器,以及这些主服务器属下的所有从服务器,并在被监视的主服务器故障时,自动将下线主服务器属下的某个从服务器升级为新的主服务器,然后由新的主服务器代替已下线的主服务器继续处理命令请求。Redis提供的sentinel(哨兵)机制,通过sentinel模式启动redis后,自动监控master/slave的运行状态,基本原理是:心跳机制+投票裁决
监控(Monitoring): Sentinel 会不断地检查你的主服务器和从服务器是否运作正常。
提醒(Notification): 当被监控的某个 Redis 服务器出现问题时, Sentinel 可以通过 API向管理员或者其他应用程序发送通知。
自动故障迁移(Automatic failover): 当一个主服务器不能正常工作时, Sentinel 会开始一次自动故障迁移操作, 它会将失效主服务器的其中一个从服务器选举出来,升级为新的主服务器, 并让失效主服务器的其他从服务器改为复制新的主服务器; 当客户端试图连接失效的主服务器时, 集群也会向客户端返回新主服务器的地址, 使得集群可以使用新主服务器代替失效服务器。
二、Sentinel的主从原理 以下为Sentinel架构以及主从切换图:
Jedis2.2.2之前版本,因为主从实例地址(IP PORT)是不同的,当故障发生进行主从切换后,应用程序无法知道新地址,故在Jedis2.2.2中新增了对Sentinel的支持,应用通过redis.clients.jedis.JedisSentinelPool.getResource()取得的Jedis实例会及时更新到新的主实例地址。
三、Redis Sentinel高可用集群搭建 首先稍微介绍下如何在linux上安装redis
redis安装 1
2
3
4
$ wget http://download.redis.io/releases/redis-3.2.8.tar.gz
$ tar xzf redis-3.2.8.tar.gz
$ cd redis-3.2.8
$ make
测试: 首先启动redis服务1
fish@test -vm:~/server/redis-3.2.8$ src/redis-server
然后打开一个新的命令窗口,启动redis client端进行测试。1
2
3
4
5
fish@test -vm:~/server/redis-3.2.8$ src/redis-cli
127.0.0.1:6379> set name fish
OK
127.0.0.1:6379> get name
"fish"
redis sentinel集群搭建 硬件条件有限,这里我将采用伪分布式进行搭建,所有节点都在同一台虚拟机,通过不同端口区分:2个哨兵,1个主redis,2个从redis 首先在redis目录创建conf目录,然后在其中添加一下配置文件,配置文件如下:
1
2
3
4
5
6
fish@test -vm:~/server/redis-3.2.8/conf$ ls -1
redis-master-6379.conf
redis-slave-6380.conf
redis-slave-6381.conf
sentinel-63791.conf
sentinel-63792.conf
redis节点配置可通过复制redis目录下的redis.conf默认配置进行相应修改,sentinel节点的配置文件则可通过复制redis目录下的sentinel.conf配置内容进行相应修改。 sentinel_63791.conf 配置:1
2
3
4
5
6
7
8
9
port 63791
daemonize yes
logfile "/var/log/sentinel_63791.log"
sentinel monitor master-1 192.168.78.99 6379 2
sentinel down-after-milliseconds master-1 5000
sentinel failover-timeout master-1 18000
sentinel auth-pass master-1 yingjun
sentinel parallel-syncs master-1 1
sentinel_63792.conf 配置:1
2
3
4
5
6
7
8
9
port 63792
daemonize yes
logfile "/var/log/sentinel_63792.log"
sentinel monitor master-1 192.168.78.99 6379 2
sentinel down-after-milliseconds master-1 5000
sentinel failover-timeout master-1 18000
sentinel auth-pass master-1 yingjun
sentinel parallel-syncs master-1 1
redis_master_6379.conf 配置: 在原配置文件中作如下修改:1
2
3
4
port 6379
daemonize yes
requirepass yingjun
masterauth yingjun
redis_slave_6380.conf 配置: 在原配置文件中作如下修改:1
2
3
4
5
port 6380
daemonize yes
requirepass yingjun
slaveof 192.168.78.99 6379
masterauth yingjun
redis_slave_6381.conf 配置: 在原配置文件中作如下修改:1
2
3
4
5
port 6381
daemonize yes
requirepass yingjun
slaveof 192.168.78.99 6379
masterauth yingjun
按如下顺序依次启动服务:1
2
3
4
5
./redis-server ../conf/redis_master_6379.conf
./redis-server ../conf/redis_slave_6381.conf
./redis-server ../conf/redis_slave_6382.conf
./redis-sentinel ../conf/sentinel_63791.conf
./redis-sentinel ../conf/sentinel_63792.conf
查看进程是否都已经启动:
1
2
3
4
5
6
7
fish@test -vm:~$ ps -ef|grep redis
fish 30233 2408 0 15:23 ? 00:00:03 src/redis-server *:6379
fish 30257 2408 0 15:24 ? 00:00:03 src/redis-server *:6380
fish 30270 2408 0 15:24 ? 00:00:03 src/redis-server *:6381
fish 30503 2408 0 16:06 ? 00:00:00 src/redis-sentinel *:63791 [sentinel]
fish 30514 2408 0 16:06 ? 00:00:00 src/redis-sentinel *:63792 [sentinel]
fish 30549 8143 0 16:09 pts/18 00:00:00 grep --color=auto redis
查看master的状态:1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
fish@test -vm:~/server/redis-3.2.8$ src/redis-cli -h 10.14.137.85 -p 6379
10.14.137.85:6379> set name tom
(error) NOAUTH Authentication required.
10.14.137.85:6379> auth fish@123
OK
10.14.137.85:6379> info replication
role:master
connected_slaves:2
slave0:ip=10.14.137.85,port=6380,state=online,offset=66732,lag=1
slave1:ip=10.14.137.85,port=6381,state=online,offset=66732,lag=1
master_repl_offset:66871
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:2
repl_backlog_histlen:66870
10.14.137.85:6379>
可以看到role为master,同时显示了旗下有两个slave 通过src/redis-cli -h 10.14.137.85 -p 6379命令登录master客户端,如何redis设置了password,此时可以进入,但是不能进行操作,需通过autho password命令授权后,在可进行其他操作。也可登录时加上-a password参数指定密码。
查看slave的状态:1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
fish@test -vm:~/server/redis-3.2.8$ src/redis-cli -h 10.14.137.85 -p 6380
10.14.137.85:6380> get name
(error) NOAUTH Authentication required.
10.14.137.85:6380> auth fish@123
OK
10.14.137.85:6380> info replication
role:slave
master_host:10.14.137.85
master_port:6379
master_link_status:up
master_last_io_seconds_ago:0
master_sync_in_progress:0
slave_repl_offset:84567
slave_priority:100
slave_read_only:1
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
10.14.137.85:6380>
可以看到role为slave,只读不能写
查看sentinel的状态:1
2
3
4
fish@test -vm:~/server/redis-3.2.8$ src/redis-cli -h 10.14.137.85 -p 63791
10.14.137.85:63791> info sentinel
DENIED Redis is running in protected mode because protected mode is enabled, no bind address was specified, no authentication password is requested to clients. In this mode connections are only accepted from the loopback interface. If you want to connect from external computers to Redis you may adopt one of the following solutions: 1) Just disable protected mode sending the command 'CONFIG SET protected-mode no' from the loopback interface by connecting to Redis from the same host the server is running, however MAKE SURE Redis is not publicly accessible from internet if you do so. Use CONFIG REWRITE to make this change permanent. 2) Alternatively you can just disable the protected mode by editing the Redis configuration file, and setting the protected mode option to 'no' , and then restarting the server. 3) If you started the server manually just for testing, restart it with the '--protected-mode no' option. 4) Setup a bind address or an authentication password. NOTE: You only need to do one of the above things in order for the server to start accepting connections from the outside.
10.14.137.85:63791>
从以上输出可以发现,竟然出错了,仔细查看输出信息,说是redis运行在保护模式 protected-mode 是为了禁止公网访问redis cache,加强redis安全的。它启用的条件,有两个:
如果启用了,则只能够通过lookback ip(127.0.0.1)访问Redis cache,如果从外网访问,则会返回相应的错误信息,就是上图中的信息。 因此在新的版本中,应该配置绑定IP和访问密码,这样的话才不会报错误 这里master及两台slave redis由于设置了pass,所以不会提示,但sentinel既没设置pass也没绑定ip,所以连接sentinel时出现以上提示,解决方案:
更改配置文件,将protected-mode设置为no
通过命令关闭保护模式:CONFIG SET protected-mode no
重启服务,启动时加上参数:–protected-mode no
绑定ip或设置pass
采取其中一种即可。 ok后,查看sentinel状态:1
2
3
4
5
6
7
8
9
10
fish@test -vm:~/server/redis-3.2.8$ src/redis-cli -h 10.14.137.85 -p 63791
10.14.137.85:63791> info sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=master-1,status=ok,address=10.14.137.85:6379,slaves=2,sentinels=2
10.14.137.85:63791>
接下来验证redis sentinel的主从切换:
首先关闭主redis(6379)服务(shutdown)。 查看哨兵,发现端口号为6380的从服务变成了主服务,sentinel自动完成了故障切换。1
2
3
4
5
6
7
8
9
10
11
fish@test -vm:~/server/redis-3.2.8$ kill -9 31013
fish@test -vm:~/server/redis-3.2.8$ src/redis-cli -h 10.14.137.85 -p 63791
10.14.137.85:63791> info sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=master-1,status=ok,address=10.14.137.85:6380,slaves=2,sentinels=2
10.14.137.85:63791>
可以看到哨兵所监听的master address变成了address=10.14.137.85:6380
启动刚才被shutdown的6379服务并查看,发现它变成了从服务。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
fish@test -vm:~/server/redis-3.2.8$ src/redis-cli -h 10.14.137.85 -p 6380
10.14.137.85:6380> info replication
NOAUTH Authentication required.
10.14.137.85:6380> auth fish@123
OK
10.14.137.85:6380> info replication
role:master
connected_slaves:1
slave0:ip=10.14.137.85,port=6381,state=online,offset=22085,lag=0
master_repl_offset:22085
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:2
repl_backlog_histlen:22084
10.14.137.85:6380>
可见,之前的slave变成了master 重新启动之前的master:1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
fish@test -vm:~/server/redis-3.2.8$ src/redis-server conf/redis-master-6379.conf
fish@test -vm:~/server/redis-3.2.8$ src/redis-cli -h 10.14.137.85 -p 6379 -a fish@123
10.14.137.85:6379> info replication
role:slave
master_host:10.14.137.85
master_port:6380
master_link_status:up
master_last_io_seconds_ago:1
master_sync_in_progress:0
slave_repl_offset:46407
slave_priority:100
slave_read_only:1
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
10.14.137.85:6379>
发现之前的master下线重启之后role变为了slave。
四、Jedis Sentinel教程 Maven依赖:1
2
3
4
5
6
7
8
9
10
11
<dependency >
<groupId > redis.clients</groupId >
<artifactId > jedis</artifactId >
<version > 2.8.0</version >
</dependency >
<dependency >
<groupId > org.springframework.data</groupId >
<artifactId > spring-data-redis</artifactId >
<version > 1.6.4.RELEASE</version >
</dependency >
redis的配置文件:1
2
3
4
5
6
7
8
9
10
11
12
redis.pass=yingjun
redis.pool.maxTotal=105
redis.pool.maxIdle=10
redis.pool.maxWaitMillis=60000
redis.pool.testOnBorrow=true
sentinel1.ip=192.168.78.99
sentinel1.port=63791
sentinel2.ip=192.168.78.99
sentinel2.port=63792
Spring的配置文件:1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
<bean id ="jedisPoolConfig" class ="redis.clients.jedis.JedisPoolConfig" >
<property name ="maxTotal" value ="${redis.pool.maxTotal}" />
<property name ="maxIdle" value ="${redis.pool.maxIdle}" />
<property name ="maxWaitMillis" value ="${redis.pool.maxWaitMillis}" />
<property name ="testOnBorrow" value ="${redis.pool.testOnBorrow}" />
</bean >
<bean id ="sentinelConfiguration" class ="org.springframework.data.redis.connection.RedisSentinelConfiguration" >
<property name ="master" >
<bean class ="org.springframework.data.redis.connection.RedisNode" >
<property name ="name" value ="master-1" />
</bean >
</property >
<property name ="sentinels" >
<set >
<bean class ="org.springframework.data.redis.connection.RedisNode" >
<constructor-arg name ="host" value ="${sentinel1.ip}" > </constructor-arg >
<constructor-arg name ="port" value ="${sentinel1.port}" > </constructor-arg >
</bean >
<bean class ="org.springframework.data.redis.connection.RedisNode" >
<constructor-arg name ="host" value ="${sentinel2.ip}" > </constructor-arg >
<constructor-arg name ="port" value ="${sentinel2.port}" > </constructor-arg >
</bean >
</set >
</property >
</bean >
<bean id ="jedisConnectionFactory" class ="org.springframework.data.redis.connection.jedis.JedisConnectionFactory" >
<property name ="password" value ="${redis.pass}" />
<property name ="poolConfig" >
<ref bean ="jedisPoolConfig" />
</property >
<constructor-arg name ="sentinelConfig" ref ="sentinelConfiguration" />
</bean >
<bean id ="redisTemplate" class ="org.springframework.data.redis.core.RedisTemplate" >
<property name ="connectionFactory" ref ="jedisConnectionFactory" />
<property name ="keySerializer" >
<bean class ="org.springframework.data.redis.serializer.StringRedisSerializer" />
</property >
<property name ="hashKeySerializer" >
<bean class ="org.springframework.data.redis.serializer.StringRedisSerializer" />
</property >
<property name ="valueSerializer" >
<bean class ="org.springframework.data.redis.serializer.JdkSerializationRedisSerializer" />
</property >
<property name ="hashValueSerializer" >
<bean class ="org.springframework.data.redis.serializer.JdkSerializationRedisSerializer" />
</property >
</bean >
<bean id ="stringRedisTemplate" class ="org.springframework.data.redis.core.StringRedisTemplate" >
<property name ="connectionFactory" ref ="jedisConnectionFactory" />
</bean >
</beans >
代码中直接用redisTemplate调用:1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
@Override
public boolean add (final KeyToken tkey) {
boolean result = redisTemplate.execute(new RedisCallback<Boolean>() {
@Override
public Boolean doInRedis (RedisConnection connection) throws DataAccessException {
RedisSerializer<String> serializer = getRedisSerializer();
byte [] key = serializer.serialize(tkey.getIndex());
byte [] name = serializer.serialize(tkey.getExpire_time());
return connection.setNX(key, name);
}
});
return result;
}
以下是测试代码:1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
package com.zxy.lab.code.redis;
import com.google.gson.Gson;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.dao.DataAccessException;
import org.springframework.data.redis.connection.RedisConnection;
import org.springframework.data.redis.core.RedisCallback;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.data.redis.core.StringRedisTemplate;
import org.springframework.data.redis.core.ValueOperations;
import org.springframework.data.redis.serializer.RedisSerializer;
import org.springframework.test.context.ContextConfiguration;
import org.springframework.test.context.junit4.SpringJUnit4ClassRunner;
import javax.annotation.Resource;
import java.io.Serializable;
import java.util.Date;
* Created by cdzhouxiaoyu@jd.com on 2017/3/28.
*/
@RunWith (SpringJUnit4ClassRunner.class)
@ContextConfiguration ("classpath:spring-config.xml" )
public class SentinelRedisTest {
@Resource (name = "stringRedisTemplate" )
private StringRedisTemplate stringRedisTemplate;
@Autowired
private RedisTemplate<String, Object> redisTemplate;
@Test
public void testPutCache () {
String key = "redis-test" ;
ValueOperations<String, String> valueOperations = stringRedisTemplate.opsForValue();
valueOperations.set(key, "hello" );
System.out.println(valueOperations.get(key));
}
@Test
public void testPutObject () {
String key = "redis-test-order" ;
Order order = new Order(1234L , 100 , new Date());
ValueOperations<String, Object> valueOperations = redisTemplate.opsForValue();
valueOperations.set(key, order);
System.out.println(valueOperations.get(key));
}
@Test
public void testPutObjectByStringRedisTemplate () {
final String key = "redis-test-order" ;
final Order order = new Order(1234L , 100 , new Date());
Boolean result = stringRedisTemplate.execute(new RedisCallback<Boolean>() {
@Override
public Boolean doInRedis (RedisConnection redisConnection) throws DataAccessException {
RedisSerializer<String> stringSerializer = redisTemplate.getStringSerializer();
byte [] k = stringSerializer.serialize(key);
byte [] v = stringSerializer.serialize(new Gson().toJson(order));
return redisConnection.setNX(k, v);
}
});
Object value = stringRedisTemplate.execute(new RedisCallback<Object>() {
@Override
public Object doInRedis (RedisConnection redisConnection) throws DataAccessException {
RedisSerializer<String> stringSerializer = redisTemplate.getStringSerializer();
byte [] v = redisConnection.get(stringSerializer.serialize(key));
String valueStr = stringSerializer.deserialize(v);
return new Gson().fromJson(valueStr, Order.class);
}
});
System.out.println(value);
}
public static class Order implements Serializable {
private Long orderId;
private double price;
private Date submitTime;
public Order () {
}
public Order (Long orderId, double price, Date submitTime) {
this .orderId = orderId;
this .price = price;
this .submitTime = submitTime;
}
public Long getOrderId () {
return orderId;
}
public void setOrderId (Long orderId) {
this .orderId = orderId;
}
public double getPrice () {
return price;
}
public void setPrice (double price) {
this .price = price;
}
public Date getSubmitTime () {
return submitTime;
}
public void setSubmitTime (Date submitTime) {
this .submitTime = submitTime;
}
@Override
public String toString () {
return "Order{" +
"orderId=" + orderId +
", price=" + price +
", submitTime=" + submitTime +
'}' ;
}
}
}
常见问题 以下是测试过程中遇到的问题:
测试工程启动时报如下错误:
java.lang.NoSuchMethodError: org.springframework.core.serializer.support.DeserializingConverter
详细错误信息如下:1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'redisTemplate' defined in class path resource [spring/spring-redis.xml]: Invocation of init method failed; nested exception is java.lang.NoSuchMethodError: org.springframework.core.serializer.support.DeserializingConverter.<init>(Ljava/lang/ClassLoader;)V
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1514)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:519)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:293)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:223)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:290)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:191)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:638)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:942)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:482)
at org.springframework.test.context.support.AbstractGenericContextLoader.loadContext(AbstractGenericContextLoader.java:120)
at org.springframework.test.context.support.AbstractGenericContextLoader.loadContext(AbstractGenericContextLoader.java:60)
at org.springframework.test.context.support.AbstractDelegatingSmartContextLoader.delegateLoading(AbstractDelegatingSmartContextLoader.java:102)
at org.springframework.test.context.support.AbstractDelegatingSmartContextLoader.loadContext(AbstractDelegatingSmartContextLoader.java:246)
at org.springframework.test.context.CacheAwareContextLoaderDelegate.loadContextInternal(CacheAwareContextLoaderDelegate.java:69)
at org.springframework.test.context.CacheAwareContextLoaderDelegate.loadContext(CacheAwareContextLoaderDelegate.java:95)
... 29 more
Caused by: java.lang.NoSuchMethodError: org.springframework.core.serializer.support.DeserializingConverter.<init>(Ljava/lang/ClassLoader;)V
at org.springframework.data.redis.serializer.JdkSerializationRedisSerializer.<init>(JdkSerializationRedisSerializer.java:54)
at org.springframework.data.redis.core.RedisTemplate.afterPropertiesSet(RedisTemplate.java:122)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1573)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1511)
... 44 more
原因: spring版本过低,spring-data-redis采用了新的构造函数。
解决方案: 升级spring版本指4.2.1以上